Video gaming console that synchronizes digital images with variations in musical tempo

ABSTRACT

The teachings described herein are generally directed to a system, method, and apparatus for separating and mixing tracks within music. The system can have a video that is synchronized with the variations in the musical tempo through a variable timing reference track designed and provided for a user of the preselected performance that was prerecorded, wherein the designing of the variable timing reference track includes creating a tempo map having variable tempos, rhythms, and beats using notes from the preselected performance.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/229,973, filed Dec. 21, 2018, which is a continuation of U.S.application Ser. No. 15/446,428, filed Mar. 1, 2017, now U.S. Pat. No.10,192,460, issued Jan. 29, 2019, which is a continuation of U.S.application Ser. No. 15/062,165, filed Mar. 6, 2016, now U.S. Pat. No.9,626,877, issued Apr. 18, 2017, which is a continuation of U.S.application Ser. No. 13/886,269, filed May 2, 2013, now U.S. Pat. No.9,311,824, issued Apr. 12, 2016, which is a continuation of U.S.application Ser. No. 13/115,937, filed May 25, 2011, now U.S. Pat. No.8,476,517, issued Jul. 2, 2013, which is a continuation of Ser. No.13/023,485, filed Feb. 8, 2011, now U.S. Pat. No. 8,207,438, issued Jun.26, 2012, which is a continuation of U.S. application Ser. No.12/390,393, filed Feb. 20, 2009, now U.S. Pat. No. 7,902,446, issuedMar. 8, 2011, and claims the benefit of U.S. Provisional Application No.61/030,174, filed Feb. 20, 2008, each application of which is herebyincorporated herein by reference in its entirety.

BACKGROUND Field of the Invention

The teachings generally relate to a system for learning and mixingmusic, the system having a processor, a custom digital audio filedatabase, a transformation module, an emulation recording module, anintegration engine, an input device to record music, and output devicesthat include a graphical user interface and a speaker to deliver musicto a user.

Description of the Related Art

Multi-track recording technology provides flexibility to record music asindividual audio tracks, either together or independently, and then mixthe audio tracks to produce a desired compilation of music fordistribution. The distributed versions will normally have a reducednumber of tracks: a single track in the case of mono-sound, two tracksfor stereo sound and typically six tracks for a surround sound system.When the music is compiled to have a reduced number of tracks, themusical parts are combined in such a way that accessing individualinstrument contributions becomes difficult-to-impossible.

Musicians and singers have a longfelt-but-unsolved need to clearlyisolate a musical instrument track from a preselected piece of music,whether that musical instrument comprises the vocal cords of a humanbeing, a brass instrument in big band, a string instrument, a percussioninstrument, or any other musical instrument known to one of skill. And,currently, there is no way to hear all of the articulations and detailsin a performance that a user desires to learn, particularly when theperformance is in an embedded mix of music. This is particularly aproblem for users that cannot read music. Unfortunately, current methodsof separating sounds through equalization or the use of algorithms tomask other sounds or pull out particular frequencies have beeninsufficient to address the need. The isolation of a musical instrumenttrack from the preselected music would allow one to emulate thepreselected piece of music on a preselected musical instrument in aprocess of learning the music. Musicians currently attempt to play alongwith a recording of a musical compilation. The problem, however, is thatthe musician finds it difficult to clearly identify all aspects of thepreselected piece of music as it is embedded in a musical compilation.

Musicians can also obtain a modified musical compilation having amusical instrument removed from it, and the musician can play along withthe modified musical compilation. The problem, however, is that themusician cannot listen to the musical instrument alone with the detailsthat are contributed by the musical instrument to the musicalcompilation. Moreover, the musician does not have the ability to recordand mix the musician's performance with the prerecorded portions of thework, so that the musician can self-critique the performance to enhancelearning.

Accordingly, one of skill will appreciate a system that allows a user to(1) obtain a custom digital audio file of a preselected piece of music,(2) transform the custom digital audio file into an isolated instrumentaudio track or an emulation audio track, (3) emulate the preselectedpiece of music with a preselected musical instrument, (4) record anemulated instrument audio track, combine the emulated instrument audiotrack with the emulation audio track to transform the custom digitalaudio file into an education audio track, (5) listen to the educationalaudio track to identify deficiencies in the accuracy of the emulating,and (6) repeat the emulating, recording, combining, and listening untilthe preselected piece of music has been learned on the preselectedmusical instrument. Moreover, a system that is interactive and fun touse will make the learning experience more rewarding to the user and thesystem more attractive to the marketplace. Such a system will address alongfelt and unsolved need of musicians and vocalists, whetheraccomplished or aspiring.

SUMMARY

The teachings described herein are generally directed to a system forlearning music through an educational audio track embodied on a computerreadable medium. The system can comprise components including aprocessor, an input device, a database, a transformation module, anemulation recording module, an integration engine, an output module, andan output device, wherein each component is operable in itself toperform its function in the system and operable with other systemcomponents to provide a system to a user for learning music.

In some embodiments, the teachings are directed to a method of learningmusic through an educational audio track. The method includes obtaininga multi-track digital audio file produced from a multi-track digitalaudio recording. The multi-track digital audio file comprises anisolated instrument audio track and an emulation audio track, and theisolated instrument audio track comprises a single musical instrumentplaying a preselected piece of music that a user desires to learn on apreselected musical instrument. The method includes transforming thecomposition of a multi-track digital audio file to include a ratio of(i) the isolated instrument audio track to (ii) an emulation audiotrack. The emulation audio track represents a subtraction of theisolated instrument audio track from the plurality of audio tracks, andthe gain ratio is selected by the user. The method includes emulatingthe preselected piece of music by listening to the isolated instrumentaudio track and playing the preselected musical instrument to create anemulated instrument audio track. The emulated instrument audio track isrecorded on a computer readable medium and combined with the emulationaudio track to transform the custom digital audio file into aneducational audio file. The method includes listening to the educationalaudio track to identify deficiencies in the emulating by the user. Theuser repeats the emulating, recording, combining, and listening untilthe user has learned the preselected piece of music on the preselectedmusical instrument to the user's satisfaction.

In some embodiments, the transforming includes reducing the volume ofthe emulation audio track and, in some embodiments, the transformingincludes reducing the volume of the isolated instrument audio track.

The method can further comprise selecting one or more bars of theisolated instrument audio track to enable the user to focus on emulatinga section of the preselected piece of music. In some embodiments, themethod can include looping the selection one or more bars to provide arepeated playback of the section.

In some embodiments, the emulating can further comprise reading adigital musical transcription and tablature display corresponding to theisolated instrument audio track. And, in some embodiments, the customdigital audio file further comprises an isolated metronome audio track,and the method further comprises listening to the isolated metronomeaudio track that is designed for the preselected piece of music.

The teachings include a system comprising a processor, an input deviceoperable to receive audio data on a computer readable medium, a databaseoperable to store audio files on a computer readable medium for access,and a transformation module embodied in a computer readable medium. Insome embodiments, the transformation module is operable to transform amulti-track digital audio file comprising an isolated instrument audiotrack and an emulation audio track into a ratio of (i) the isolatedinstrument audio track to (ii) the emulation audio track. The emulationaudio track represents a subtraction of the isolated instrument audiotrack from the plurality of audio tracks, and the transforming canresult from a user selecting a gain ratio between the isolatedinstrument audio track, the emulation audio track, and the metronometrack. The system can also include an emulation recording moduleembodied in a computer readable medium. The emulation recording moduleis operable to record the user's emulated audio track on a computerreadable medium. The system can also include an integration engineembodied in a computer readable medium, wherein the integration engineis operable to combine the emulated instrument audio track with theemulation audio track to transform the multi-track digital audio fileinto an educational audio file. In addition, the system can include anoutput module embodied in a computer readable medium, wherein the outputmodule is operable to transmit audio data to an output device. Theoutput device can be operable to provide audio data to the user, whereinthe audio data assists the user in learning a preselected piece ofmusic.

In some embodiments, the input device comprises a microphone or a lineinput. The line input can be used, for example, for the user to inputaudio data from their musical instrument into the system for learningand mixing music. For example, the microphone could receive audio from apiano, or the line input could receive output from a guitar amplifier.One of skill will appreciate that the data input can be analog ordigital, and that conversions can be where necessary.

In some embodiments, the output module transmits music transcription andtablature data to a graphical user interface. The output module can alsohave a recalibration function operable to recalibrate an audio datatrack output to correct a latency in the output of the audio track data.In some embodiments, the output device comprises a speaker, a graphicaluser interface, or both a speaker and a graphical user interface, forexample. And, in some embodiments, the output module has asynchronization function operable to synchronize the music transcriptionand tablature data display on the graphical user interface with theisolated instrument audio track provided to the listener through thespeaker.

The multi-track digital audio file can further comprise a metronomeaudio track. As such, in some embodiments, the transformation module canbe operable to transform the multi-track digital audio file into a ratioof (i) the isolated instrument audio track. (ii) the emulation audiotrack, and (iii) the metronome audio track, and a gain ratio between theisolated instrument audio track, the emulation audio track, and themetronome audio track can be selected by the user.

The system can further comprise a data exchange module embodied in acomputer readable medium, wherein the data exchange module is operableto exchange data with external computer readable media. In someembodiments, the system is contained in a hand-held device; operable tofunction as a particular machine or apparatus having the additionalfunction of telecommunications, word processing, or gaming; or operableto function as a particular machine or apparatus not having othersubstantial functions.

The teachings are also directed to a handheld apparatus for learningmusic. The apparatus can include a processor; an input device comprisinga microphone and a data input port, wherein the input device is operableto receive and store audio data on a computer readable medium; adatabase operable to store audio files on a computer readable medium foraccess; and a transformation module embodied in a computer readablemedium. The transformation module can be operable to transform amulti-track digital audio file comprising an isolated instrument audiotrack, an emulation audio track, and a metronome track into a ratio of(i) the isolated instrument audio track, (ii) the emulation audio track,and (iii) the metronome track. The emulation audio track represents asubtraction of the isolated instrument audio track from the plurality ofaudio tracks, and the transforming can result from a user selecting again ratio between the isolated instrument audio track, the emulationaudio track, and the metronome track. The apparatus includes anemulation recording module embodied in a computer readable medium,wherein the emulation recording module is operable to record the user'semulated audio track on a computer readable medium. The apparatusincludes an integration engine embodied in a computer readable medium,wherein the integration engine is operable to combine the emulatedinstrument audio track with the emulation audio track to transform themulti-track digital audio file into an educational audio file. Theapparatus includes an output module embodied in a computer readablemedium, wherein the output module is operable to transmit audio data toan output device. And, the apparatus includes an output devicecomprising a speaker and a graphical user interface. The output devicecan be operable to provide audio data to the user in the form of soundand graphics, wherein the audio data assists the user in learning apreselected piece of music.

In some embodiments, the apparatus is operable to function as aparticular machine or apparatus having the additional function oftelecommunications, word processing, or gaming; or operable to functionas a particular machine or apparatus not having other substantialfunctions.

The teachings are also directed to a metronome for measuring time inmusic, wherein the metronome comprises a processor, an input deviceoperable to receive audio data on a computer readable medium, and adatabase operable to store audio files on a computer readable medium foraccess, wherein the audio files comprise metronome audio tracks thatwere designed for a preselected piece of music. The metronome includes atransformation module embodied in a computer readable medium, whereinthe transformation module can be operable to transform a multi-trackdigital audio file comprising a preselected music track and a metronometrack into a ratio of (i) the preselected music track to (ii) themetronome track. The transforming can result from a user selecting again ratio between the preselected music track and the metronome track.The metronome includes an output module embodied in a computer readablemedium, wherein the output module is operable to transmit audio data toan output device. The output device can be operable to provide audiodata to the user, wherein the audio data assists the user in learning apreselected piece of music.

There are a variety of ways that a metronome audio track can be designedfor the preselected piece of music. In some embodiments, the metronomeaudio track can be designed using the preselected piece of music in adigital audio workstation (DAW), such as PROTOOLS (a Digidesign product)or LOGIC (an Apple product). Using the digital audio workstation, aprogrammer locates transients in the multi-track digital audio file suchas, for example, the location of each quarter note, where the quarternote is the transient. The programmer places a MIDI note on a MIDI trackto represent each transient. In some embodiments, the MIDI note isplaced wherever the transient lies, such as an eighth note, quarternote, or the like. The space between each MIDI note that was manuallycreated is calculated to determine the beats per minute of each bar ofmusic for the preselected piece if music. A tempo map is created byanalyzing the MIDI track with the manually created MIDI notes. Ametronome audio file is created by placing an audio sound, such as abell, wood block, cow bell, or any such tone on each beat of the tempomap that corresponds with the preselected piece of music. A music XMLfile is derived from the exported MIDI metronome track which is used tosynchronize the graphical notation with the metronome audio track andthe preselected piece of music.

In some embodiments, the metronome is operable to function as aparticular machine or apparatus having the additional function oftelecommunications, word processing, or gaming; or operable to functionas a particular machine or apparatus not having other substantialfunctions.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 shows a general technology platform for the system for learningand mixing music, according to some embodiments.

FIG. 2 illustrates a processor-memory diagram to describe components ofthe system for learning and mixing music, according to some embodiments.

FIG. 3 is a concept diagram illustrating the system for learning andmixing music, according to some embodiments.

FIG. 4 is a diagram of the logic of the system for learning and mixingmusic, according to some embodiments.

FIG. 5 illustrates a circuit diagram of the system for learning andmixing music, according to some embodiments.

FIG. 6 illustrates a circuit diagram of the system for learning andmixing music including a metronome function, according to someembodiments.

FIG. 7 illustrates a musical notation display designed for use with thesystem for learning and mixing music, according to some embodiments.

FIG. 8 illustrates a device for learning and mixing music designed foruse with the system for learning and mixing music, according to someembodiments.

FIG. 9 illustrates a display for a graphical user interface identifyinga multi-track audio recording set designed for use with the system forlearning and mixing music, according to some embodiments.

FIG. 10 illustrates a display for a graphical user interface offeringselections within a track list of a multi-track audio recording setdesigned for use with the system for learning and mixing music,according to some embodiments.

FIG. 11 illustrates a display for a graphical user interface offeringinformation on the track list selection in the system for learning andmixing music, according to some embodiments.

FIG. 12 illustrates a display for a graphical user interface offeringstate selections for track faders, volume fader control, and transportselections, in which all tracks are selected on a multi-track audiorecording designed for use with the system for learning and mixingmusic, according to some embodiments.

FIG. 13 illustrates a display for a graphical user interface offeringstate selections for track faders, volume fader control, and transportselections, in which all tracks but the metronome track are selected ona multi-track audio recording designed for use with the system forlearning and mixing music, according to some embodiments.

FIG. 14 illustrates a display for a graphical user interface offeringstate selections for track faders, volume fader control, and transportselections, in which only the isolated instrument audio track and theemulated instrument audio track are selected on a multi-track audiorecording designed for use with the system for learning and mixingmusic, according to some embodiments.

FIG. 15 illustrates a display for a graphical user interface offeringstate selections for track faders, volume fader control, and transportselections, in which only the emulation audio track and the emulatedinstrument audio track are selected on a multi-track audio recordingdesigned for use with the system for learning and mixing music,according to some embodiments.

FIG. 16 illustrates a display for a graphical user interface offeringmusical transcription and tablature audio data, in which only thecurrent section of music is shown for a multi-track audio recordingdesigned for use with the system for learning and mixing music,according to some embodiments.

FIG. 17 illustrates a display for a graphical user interface offeringselection of a section of a piece of music by bar or set of bars in amulti-track audio recording designed for use with the system forlearning and mixing music, according to some embodiments.

FIG. 18 illustrates a display for a graphical user interface offering ahelp page for the system for learning and mixing music, according tosome embodiments.

FIG. 19 shows how a network may be used for the system for learning andmixing music, according to some embodiments.

DETAILED DESCRIPTION OF THE INVENTION

The teachings described herein are generally directed to a system forlearning music through an educational audio track embodied on a computerreadable medium. The system can comprise components including aprocessor, an input device, a database, a transformation module, anemulation recording module, an integration engine, an output module, andan output device, wherein each component is operable in itself toperform its function in the system and operable with other systemcomponents to provide a system to a user for learning music.

The teachings include a system comprising a processor, an input deviceoperable to receive audio data on a computer readable medium, a databaseoperable to store audio files on a computer readable medium for access,and a transformation module embodied in a computer readable medium. Insome embodiments, the transformation module is operable to transform amulti-track digital audio file comprising an isolated instrument audiotrack and an emulation audio track into a ratio of (i) the isolatedinstrument audio track to (ii) the emulation audio track. The emulationaudio track represents a subtraction of the isolated instrument audiotrack from the plurality of audio tracks, and the transforming canresult from a user selecting a gain ratio between the isolatedinstrument audio track, the emulation audio track, and the metronometrack. The system can also include an emulation recording moduleembodied in a computer readable medium. The emulation recording moduleis operable to record the user's emulated audio track on a computerreadable medium. The system can also include an integration engineembodied in a computer readable medium, wherein the integration engineis operable to combine the emulated instrument audio track with theemulation audio track to transform the multi-track digital audio fileinto an educational audio file. In addition, the system can include anoutput module embodied in a computer readable medium, wherein the outputmodule is operable to transmit audio data to an output device. Theoutput device can be operable to provide audio data to the user, whereinthe audio data assists the user in learning a preselected piece ofmusic.

FIG. 1 shows a general technology platform for the system for learningand mixing music, according to some embodiments. The computer system 100may be a conventional computer system and includes a computer 105, I/Odevices 150, and a display device 155. The computer 105 can include aprocessor 120, a communications interface 125, memory 130, displaycontroller 135, non-volatile storage 140, and I/O controller 145. Thecomputer system 100 may be coupled to or include the I/O devices 150 anddisplay device 155.

The computer 105 interfaces to external systems through thecommunications interface 125, which may include a modem or networkinterface. It will be appreciated that the communications interface 125can be considered to be part of the computer system 100 or a part of thecomputer 105. The communications interface 125 can be an analog modem,isdn modem, cable modem, token ring interface, satellite transmissioninterface (e.g. “direct PC”), or other interfaces for coupling thecomputer system 100 to other computer systems. In a cellular telephone,this interface is typically a radio interface for communication with acellular network and may also include some form of cabled interface foruse with an immediately available personal computer. In a two-way pager,the communications interface 125 is typically a radio interface forcommunication with a data transmission network but may similarly includea cabled or cradled interface as well. In a personal digital assistant,the communications interface 125 typically includes a cradled or cabledinterface and may also include some form of radio interface, such as aBLUETOOTH or 802.11 interface, or a cellular radio interface, forexample.

The processor 120 may be, for example, a conventional microprocessorsuch as an Intel Pentium microprocessor or Motorola power PCmicroprocessor, a Texas Instruments digital signal processor, or acombination of such components. The memory 130 is coupled to theprocessor 120 by a bus. The memory 130 can be dynamic random accessmemory (DRAM) and can also include static ram (SRAM). The bus couplesthe processor 120 to the memory 130, also to the non-volatile storage140, to the display controller 135, and to the I/O controller 145.

The I/O devices 150 can include a keyboard, disk drives, printers, ascanner, and other input and output devices, including a mouse or otherpointing device. The display controller 135 may control in theconventional manner a display on the display device 155, which can be,for example, a cathode ray tube (CRT) or liquid crystal display (LCD).The display controller 135 and the I/O controller 145 can be implementedwith conventional well known technology, meaning that they may beintegrated together, for example.

The non-volatile storage 140 is often a FLASH memory or read-onlymemory, or some combination of the two. A magnetic hard disk, an opticaldisk, or another form of storage for large amounts of data may also beused in some embodiments, although the form factors for such devicestypically preclude installation as a permanent component in somedevices. Rather, a mass storage device on another computer is typicallyused in conjunction with the more limited storage of some devices. Someof this data is often written, by a direct memory access process, intomemory 130 during execution of software in the computer 105. One ofskill in the art will immediately recognize that the terms“machine-readable medium” or “computer-readable medium” includes anytype of storage device that is accessible by the processor 120 and alsoencompasses a carrier wave that encodes a data signal. Objects, methods,inline caches, cache states and other object-oriented components may bestored in the non-volatile storage 140, or written into memory 130during execution of, for example, an object-oriented software program.

The computer system 100 is one example of many possible differentarchitectures. For example, personal computers based on an Intelmicroprocessor often have multiple buses, one of which can be an I/O busfor the peripherals and one that directly connects the processor 120 andthe memory 130 (often referred to as a memory bus). The buses areconnected together through bridge components that perform any necessarytranslation due to differing bus protocols.

In addition, the computer system 100 can be controlled by operatingsystem software which includes a file management system, such as a diskoperating system, which is part of the operating system software. Oneexample of an operating system software with its associated filemanagement system software is the family of operating systems known asWindows CE® and Windows® from Microsoft Corporation of Redmond, Wash.,and their associated file management systems. Another example ofoperating system software with its associated file management systemsoftware is the LINUX operating system and its associated filemanagement system. Another example of an operating system software withits associated file management system software is the PALM operatingsystem and its associated file management system. The file managementsystem is typically stored in the non-volatile storage 140 and causesthe processor 120 to execute the various acts required by the operatingsystem to input and output data and to store data in memory, includingstoring files on the non-volatile storage 140. Other operating systemsmay be provided by makers of devices, and those operating systemstypically will have device-specific features which are not part ofsimilar operating systems on similar devices. Similarly, WinCE® or PALMoperating systems may be adapted to specific devices for specific devicecapabilities.

The computer system 100 may be integrated onto a single chip or set ofchips in some embodiments, and can be fitted into a small form factorfor use as a personal device. Thus, it is not uncommon for a processor,bus, onboard memory, and display/I-O controllers to all be integratedonto a single chip. Alternatively, functions may be split into severalchips with point-to-point interconnection, causing the bus to belogically apparent but not physically obvious from inspection of eitherthe actual device or related schematics.

FIG. 2 illustrates a processor-memory diagram to describe components ofthe system for learning and mixing music, according to some embodiments.The system 200 shown in FIG. 2 contains a processor 205 and a memory 210(that can include non-volatile memory), wherein the memory 210 includesan audio database 215, a transformation module 220, an emulationrecording module 225, an integration engine 230, an output module 235,and an optional video display module 240, which can also be a part ofthe output module 235. The system can further comprise an optional dataexchange module 245 embodied in a computer readable medium, wherein thedata exchange module is operable to exchange data with external computerreadable media.

The system includes an input device (not shown) operable to receiveaudio data on a computer readable medium. Examples of input devicesinclude a data exchange module operable to interact with external dataformats, voice-recognition software, a hand-held device in communicationwith the system including, but not limited to, a microphone, and thelike.

The audio database 215 is operable to store audio files for access on acomputer readable medium. In some embodiments, the system can storeoriginal multi-track audio files, copies of original multi-track audiofiles, and the like. Any audio file known to one of skill in the art canbe stored including, but not limited to sound files, text files, imagefiles, and the like. In some embodiments, the system can access any of avariety of accessible data through a data exchange module, as discussedabove.

Any audio format known to one of skill in the art can be used. In someembodiments, the audio file comprises a format that supports one audiocodec and, in some embodiments, the audio file comprises a format thatsupports multiple codecs. In some embodiments the audio file comprisesan uncompressed audio format such as, for example, WAV, AIFF, and AU. Insome embodiments, the audio file format comprises lossless compressionsuch as, FLAC, Monkey's Audio having file extension APE, WavPack havingfile extension WV, Shorten, Tom's lossless Audio Kompressor (TAK), TTA,ATRAC Advanced Lossless, Apple Lossless, and lossless WINDOWS MediaAudio (WMA). In some embodiments, the audio file format comprises lossycompression, such as MP3, Vorbis, Musepack, ATRAC, lossy WINDOWS MediaAudio (WMA) and AAC.

In some embodiments, the audio format is an uncompressed PCM audioformat, as a “.wav” for a WINDOWS computer readable media, or as a“.aiff” as a MAC OS computer readable media. In some embodiments aBroadcast Wave Format (BWF) can be used, allowing metadata to be storedin the file. In some embodiments, the audio format is a lossless audioformat, such as FLAC, WayPack, Monkey's Audio, ALAC/Apple Lossless. Insome embodiments, the lossless audio format provides a compression ratioof about 2:1. In some embodiments, the audio format is a free-and-openformat, such as wav, ogg, mpc, flac, aiff, raw, au, or mid, for example.In some embodiments, the audio format is an open file format, such asgsm, dct, vox, aac, mp4/m4a, or mmf. In some embodiments the audioformat is a proprietary format, such as mp3, wma, atrac, ra, ram, dss,msv, dvg, IVS, m4p, iklax, mxp4, and the like.

The transformation module 220 is operable to transform a multi-trackdigital audio file comprising an isolated instrument audio track and anemulation audio track into a ratio of (i) the isolated instrument audiotrack to (ii) the emulation audio track, wherein the emulation audiotrack represents a subtraction of the isolated instrument audio trackfrom the plurality of audio tracks, and the transforming can resultsfrom a user selecting a gain ratio, for example, between the isolatedinstrument audio track, the emulation audio track, and the metronometrack.

The emulation audio track represents a subtraction of the isolatedinstrument audio track from the plurality of audio tracks, and thetransforming can result from a user selecting a gain ratio between theisolated instrument audio track, the emulation audio track, and themetronome track. The system can also include an emulation recordingmodule 225 embodied in a computer readable medium. The emulationrecording module 225 is operable to record the user's emulated audiotrack on a computer readable medium. In some embodiments, the emulationrecording module 225 can be operable within a single functioning sectionof a system, such as a single page of a software application. In someembodiments, the emulation recording module 225 can be operable within aplurality of functioning sections of a system, such as in a plurality ofpages of a software application, such that the recording can occurquickly at the choosing of the user without having to move from onesection of the system to another section of the system.

The system can also include an integration engine 230 embodied in acomputer readable medium, wherein the integration engine 230 is operableto combine the emulated instrument audio track with the emulation audiotrack to transform the multi-track digital audio file into aneducational audio file. In addition, the system can include an outputmodule 235 embodied in a computer readable medium, wherein the outputmodule 235 is operable to transmit audio data to an output device, whichcan be a graphical user interface, or video display, which canoptionally be supported by a separate video display module 240, or thedisplay can be supported with one or more other output devices by theoutput module 235. The output device can be operable to provide audiodata to the user, wherein the audio data assists the user in learning apreselected piece of music.

In some embodiments, the input device comprises a microphone and, insome embodiments, the output module 235 transmits music transcriptionand tablature data to a graphical user interface. In some embodiments,the output device comprises a speaker, a graphical user interface, orboth a speaker and a graphical user interface, for example. And, in someembodiments, the output module has a synchronization function operableto synchronize the music transcription and tablature data display on thegraphical user interface with the isolated instrument audio trackprovided to the listener through the speaker.

The output module 235 can also have a recalibration function operable torecalibrate an audio data track output to correct a latency in theoutput of the audio track data. One of skill will appreciate that timestamps can be used to align, recalibrate, and correct latencies in theoutput of a data stream flow. In some embodiments, the latency iscorrected by time stamping samples of the audio data, where a “sample”is a short wave form of the audio having a length measured by anincrement of time. In some embodiments, a sample is less than a secondlong, (e.g., about 1/100 or 1/1000 of a seconds long). In someembodiments, the samples can be about 44/1000 of a second long. An audiotrack can contain, for example, about 44000 samples per track per secondin a high quality sound file. As such, the concept of a sample andsample resolution is a measure of audio resolution or quality. A lowerquality mono audio file has about 22000 samples per track per second,for example.

Recalibration techniques can be used in some embodiments. Bandwidthlimitations of a computer system, for example, can create variations ortransients in misalignments between tracks and hinder sound quality. Acomputer having a smaller CPU than another computer having a larger CPUbut similar memory capacity can have latency problems that result inperformance problems In some embodiments, the system can provide anoutput of 4 streaming files, and these files can include (i) anemulation audio track that does not contain the isolated instrumentaudio track, (ii) an isolated instrument audio track, (iii) an emulatedinstrument audio track, and (iv) a metronome audio track. In someembodiments, the emulation audio track, the isolated instrument audiotrack, and the emulated instrument audio track are stereo files and, insome embodiments, the metronome audio track is a mono file. Each trackin each file has it's own timeline, and there can be acceptablevariances that are not noticeable to the ear, but there can also beunacceptable variances that result in an unacceptable and audibledegradation in audio quality.

Each input file is comprised of an array of samples, and each sample canbe used as a marker in time, since each sample position has an actualposition that will serve as a measure of variance against an idealposition. The recalibration is performed on sets of samples. When agroup of samples is off in time, the system can be designed to correctfor the next set of samples. For example, a system can be designed torecalibrate based on a sample resolution that provides a 0.001 secondaccuracy by measuring the variance of a time stamp on a group of 44samples (0.001 of a second for a 44000 sample/sec high quality sample)to an ideal time stamp for that sample set. A fast method ofrecalibration was developed to reduce this variance or “latency” in theaudio so that it's not detectable to human being. A good ear, forexample, can hear time variance between tracks of about 1/60 of asecond, and multiple events of variances in a closely related string ofsamples can be cumulative, making it necessary to have a variance atleast an order of magnitude under 1/60 of a second. In some embodiments,the minimum audio resolution in a sample for latency correction shouldbe no less than 300 samples in a segment. In some embodiments, the audioresolution in a sample for latency correction is about 44 samples in asegment. In some embodiments, it was found that the recalibration shouldbe done on each “run through” of the data in the system, where a “runthrough” is an emptying and a filling of a data queue in the system.Between data loads in a queue, the recalibration occurs by measuring thedifference between the actual time of each track and the ideal time ofeach track and a correction is applied between data loads. In someembodiments, the audio queues up several hundred times per second.

The CPU on a handheld computer system can have difficulties concurrentlyprocessing the audio data files described herein. In some embodiments, ahandheld computing system may have latency difficulties whenconcurrently processing more than 2 audio data files. As such, datafiles may require compression. In some embodiments, the data files canbe compressed using a compression technique, for example, such asQUICKTIME by Apple. Other file compression techniques can be used. IMA4can also be used to compress the files in some embodiments. In someembodiments, the system requires at least a 600-700 MHz processor. TheiPhone has a 400 MHz processor, on the other hand, suggesting thatcompressed audio data files may be needed for use of some embodiments ofthe system on the iPhone. The IMA4 compression method compresses theaudio data file to about 25% of file size.

In some embodiments, it should be appreciated, however, that the systemcan use pure, uncompressed wave files. Most home PCs, however, will notneed compressed files due to the more powerful processors currentlyavailable for home PCs. The bandwidth of the computer system, i.e. thesize of the CPU and memory will dictate whether compression isnecessary. One of skill in the art will appreciate that certaincompression technologies may be needed in some systems for optimumperformance and that these technologies are readily identifiable andaccessible.

One of skill will appreciate that time stamping of data samples can alsobe used to synchronize between other data streams. In some embodiments,an additional audio data stream is used to provide a digital musicaltranscription and tablature display in the form of a graphical display.This audio data can be synchronized and recalibrated at the same time asthe other audio data.

The multi-track digital audio file can further comprise a metronomeaudio track. As such, in some embodiments, the transformation module 220can be operable to transform the multi-track digital audio file into aratio of (i) the isolated instrument audio track, (ii) the emulationaudio track, and (iii) the metronome audio track, and a gain ratiobetween the isolated instrument audio track, the emulation audio track,and the metronome audio track can be selected by the user.

As described above, the system can further comprise an optional dataexchange module 245 embodied in a computer readable medium, wherein thedata exchange module is operable to exchange data with external computerreadable media. The data exchange module can, for example, serve as amessaging module operable to allow users to communicate with other usershaving like subject-profiles, or others users in a profile independentmanner, merely upon election of the user. The users can email oneanother, post blogs, or have instant messaging capability for real-timecommunications. In some embodiments, the users have video and audiocapability in the communications, wherein the system implements datastreaming methods known to those of skill in the art. In someembodiments, the system is contained in a hand-held device; operable tofunction as a particular machine or apparatus having the additionalfunction of telecommunications, word processing, or gaming; or operableto function as a particular machine or apparatus not having othersubstantial functions.

The systems taught herein can be practiced with a variety of systemconfigurations, including personal computers, multiprocessor systems,microprocessor-based or programmable consumer electronics, network PCs,minicomputers, mainframe computers, and the like. The invention can alsobe practiced in distributed computing environments where tasks areperformed by remote processing devices that are linked through acommunications network. As such, in some embodiments, the system furthercomprises an external computer connection through the data exchangemodule 245 and a browser program module (not shown). The browser programmodule (not shown) can be operable to access external data as a part ofthe data exchange module 245.

FIG. 3 is a concept diagram illustrating the system for learning andmixing music, according to some embodiments. The system 300 containscomponents that can be used in a typical embodiment. In addition to theaudio database 215, the transformation module 220, the emulationrecording module 225, the integration engine 230, and the output module235 shown in FIG. 2, the memory 210 of the device 300 also includes adata exchange module 245 and the browser program module (not shown) foraccessing the external data. The system includes a speaker 352, display353, and a printer 354 connected directly or through I/O device 350,which is connected to I/O backplane 340.

The system 300 can be implemented in a stand-alone device, rather than acomputer system or network. In FIG. 3, for example, the I/O device 350connects to the speaker (spkr) 352, display 353, and microphone (mic)354, but could also be coupled to other features. Such a device can havea music state selector 341, an isolated instrument audio track stateselector 342, an emulation audio track state selector 343, a user'semulated audio track state selector 344, a learning state selector 345for the educational audio track, a bar state selector 346, and a timerstate selector 347 for the metronome audio track, with each stateselector connected directly to the I/O backplane 340.

In some embodiments, the system further comprises security measures toprotect the subject's privacy, integrity of data, or both. Such securitymeasures are those well-known in the art such as firewalls, software,and the like. In addition, the system can be configured for use in anenvironment that requires administrative procedures and control. Forexample, the system can include an administrative module (not shown)operable to control access, configure the engines, monitor results,perform quality assurance tests, and define audiences for targeting andtrending. Since the system can safely be provided by a network and, insome embodiments, the system is coupled to a network, the securitymeasures can help protect the contents of the system from externalintrusions.

In some embodiments, the system is a web enabled application and canuse, for example, Hypertext Transfer Protocol (HTTP) and HypertextTransfer Protocol over Secure Socket Layer (HTTPS). These protocolsprovide a rich experience for the end user by utilizing web 2.0technologies, such as AJAX, Macromedia Flash, etc. In some embodiments,the system is compatible with Internet Browsers, such as InternetExplorer, Mozilla Firefox, Opera, Safari, etc. In some embodiments, thesystem is compatible with mobile devices having full HTTP/HTTPS support,such as iPhone, PocketPCs, Microsoft Surface, Video Gaming Consoles, andthe like. In some embodiments, the system can be accessed using aWireless Application Protocol (WAP). This protocol will serve the nonHTTP enabled mobile devices, such as Cell Phones, BlackBerries, etc.,and provides a simple interface. Due to protocol limitations, the Flashanimations are disabled and replaced with Text/Graphic menus. In someembodiments, the system can be accessed using a Simple Object AccessProtocol (SOAP) and Extensible Markup Language (XML). By exposing thedata via SOAP and XML, the system provides flexibility for third partyand customized applications to query and interact with the system's coredatabases. For example, custom applications could be developed to runnatively on iPhones, Java or .Net-enabled platforms, etc. One of skillwill appreciate that the system is not limited to any of the platformsdiscussed above and will be amenable to new platforms as they develop.

FIG. 4 is a diagram of the logic of the system for learning and mixingmusic, according to some embodiments. In some embodiments, the teachingsare directed to a method of learning music through an educational audiotrack. The method includes obtaining 405 a multi-track digital audiofile produced from a multi-track digital audio recording. Themulti-track digital audio file comprises an isolated instrument audiotrack and an emulation audio track, and the isolated instrument audiotrack comprises a single musical instrument playing a preselected pieceof music that a user desires to learn on a preselected musicalinstrument. The method includes transforming 410 the composition of amulti-track digital audio file to include a ratio of (i) the isolatedinstrument audio track to (ii) an emulation audio track. The emulationaudio track represents a subtraction of the isolated instrument audiotrack from the plurality of audio tracks, and the gain ratio is selectedby the user. The method includes emulating 415 the preselected piece ofmusic by listening to the isolated instrument audio track and playingthe preselected musical instrument to create an emulated instrumentaudio track. The emulated instrument audio track is recorded 420 on acomputer readable medium and combined 425 with the emulation audio trackto transform the custom digital audio file into an educational audiofile. The method includes listening 430 to the educational audio trackto identify deficiencies in the emulating by the user. The user repeats435 the emulating, recording, combining, and listening until the userhas learned the preselected piece of music on the preselected musicalinstrument to the user's satisfaction.

In some embodiments, the transforming 410 includes reducing the volumeof the emulation audio track and, in some embodiments, the transforming410 includes reducing the volume of the isolated instrument audio track.

The method can further comprise selecting one or more bars of theisolated instrument audio track to enable the user to focus on emulatinga section of the preselected piece of music. In some embodiments, themethod can include looping the selection one or more bars to provide arepeated playback of the section.

In some embodiments, the emulating 415 can further comprise reading adigital musical transcription and tablature display corresponding to theisolated instrument audio track. And, in some embodiments, the customdigital audio file further comprises an isolated metronome audio track,and the method further comprises listening to the isolated metronomeaudio track that is designed for the preselected piece of music.

FIG. 5 illustrates a circuit diagram of the system for learning andmixing music, according to some embodiments. The solo audio part (thepart to be learned) is retained in an audio store 505, and the audiorecording of the other parts of the composition are retained in an audiostore 510. Storage areas 505 and 510 may reside in separate devices orin a single storage device, but can be accessed separately. Theprogressive retrieval of audio information from stores 505 and 510 iscontrolled by a playback controller 515 such that the various parts ofthe composition become synchronized in time. The solo audio signal canpass through a variable gain element 520 so that it's volume level in anoverall output 540 can be controlled. Similarly, the other audio signalspass through a gain element 525 and are fed to a mixing element 530, tobe combined with the solo instrument audio signal provided from gainelement 520.

An external instrument input element 545 enables an external audiosource, such as the practice instrument, to be included in overalloutput 540. The signal provided by instrument input element 545 passesthrough a gain element 550 before delivery to mixing element 530.

The overall output level can be controlled using a gain element 535,which receives an input signal from mixing element 530 and feeds anoutput signal to overall output 540. The various gain elements can becontrolled directly through user controls or through signals from acontrolling device such as a microprocessor. In some embodiments, otherparts of the musical performance may be stored in separated storageareas to facilitate the learning process or to accommodate multipleplayers.

FIG. 6 illustrates a circuit diagram of the system for learning andmixing music including a metronome function, according to someembodiments. An audible timing reference is included into the signalprovided to the output of the device. This timing reference may be aperiodic sound such as the click of a metronome, which can be recordedand stored in a method similar to that used for the other audio signals,or which may be synthesized by an audio signal generator 605 while themusical piece is being played. This additional sound is delivered tomixing element 530 via a variable gain element 610, which allows theaudible level of the timing reference to be adjusted.

FIG. 7 illustrates a musical notation display designed for use with thesystem for learning and mixing music, according to some embodiments. Insome embodiments, a display provides an animated graphicalrepresentation of the musical notation relative to the part to belearned, whereby such graphical representation scrolls along the displayarea in synchronism with the audio music signal. In particular, a timeregion 705,710 of the display area are marked to identify the presentmoment, with the graphical representation of the music moving past timeregion 705,710 as the music part is being played. By this arrangement,the display area enables a player to see musical events in advance thatare immediately following the present moment. In some embodiments, arecent musical notation may also be seen in retrospect. Accordingly, theapparatus and methods of presenting musical notation are advantageousfor at least the reason that a user is provided with a temporal contextfor the musical events of interest. Moreover, and contrary totraditional sheet music notation, the scrolling of the music can becontinuous, removing the need for page turns.

FIG. 8 illustrates a device for learning and mixing music designed foruse with the system for learning and mixing music, according to someembodiments. The device can be configured for use with any musicalinstrument, for example, guitar, piano, drums, or a vocals. As shown inFIG. 8, a device 800 can be configured for use with a piano. The device800 includes a housing or container 805, which can be of any shape, suchas a shape designed to sit on top of a piano in place of standard sheetmusic. A music display 810, which may be a liquid crystal display (LCD)screen or other type of display screen, and one or more transportbuttons 815 such as, for example, a play button, a stop button, and apause button, can be provided.

The device 800 can comprise a number of controls, which may beconfigured as knobs or other similar state selectors known in the art.In FIG. 8, a first group of state selectors 820 relate to the “blender”function of the device 800 and are structured to control the user'sinput, the music without the piano contribution, and the piano itself. Asecond group of state selectors 825 relate to the “master” function ofthe device 800 and control volume, click (or metronome signal), andtempo. A third group of state selectors 801,830 control the on/offfunction of the device 800 and may include indicator lights, lightintensity control, and additional playback controls, for example. Thedevice 800 includes one or more speakers 835, a sound module forelectric pianos (not shown), and one or more ports 802,803,840 forconnecting the device 800 to other units such as, for example, USBports, phono jacks, and power jacks, or perhaps musical instruments,such as electric organs and guitars, for example. In some embodiments, aUSB port 840 may be used to connect the device 800 to a computer system.In some embodiments, for example, USB port 840 allows for downloading ofaudio data to a larger computer memory storage location. In someembodiments, data may also be supplied to device 800 and and/or storedin removable data memory cards. Wireless studio grade headphones mayalso be provided to the player. Such devices can be produced andconfigured for ease of use with any of a variety of instruments.

In some embodiments, the multi-track digital audio files are producedfrom original multi-track digital recordings, and these recordings mayoriginate on analog tape, such as analog multi-track tape (e.g. 1 trackto 24 tracks), a digital tape format (e.g. pulse code modulation, PCM,digital tape format). In some embodiments, an analog tape format isfirst transformed into a digital recording and a multi-track digitalaudio file is produced from the digital recording. In some embodiments,the original mix is recreated by taking all of the different tracks andmaking the mix to simulate the original recording. The mixing can be amanual process and can be done with an analog console, new digitalconsole, or the mix can be done on a computer using basically any mixingtechnique known to one of skill. In some embodiments, older analog tapesneed to be restored, such as by a baking procedure, before attempting arecreation.

It should be appreciated that the teachings can apply to any piece ofmusic containing virtually any musical instrument including, but notlimited to string instruments, brass instruments, woodwind instruments,percussion instruments, and vocals. In some embodiments, pieces of musichaving variable tempos, rhythms, and beats can be learned with more easedue to the manually created and variable metronome function, as well asthe manually created audio files having superior isolated instrumentaudio track quality. In some embodiments, songs are complicated and havechanging or otherwise unisolatable beats that would be more difficult tolearn without the teachings provided herein.

Any of a variety of devices having any of a variety of graphicaldisplays can be used. And, the graphical displays can have click andslide functions for state selectors, for example, rather than knobs orphysical state selectors, such displays are depicted in FIGS. 9-18. FIG.9 illustrates a display for a graphical user interface identifying amulti-track audio recording set designed for use with the system forlearning and mixing music, according to some embodiments. Display 900represents the opening page of a multi-track digital audio file producedfrom a multi-track audio recording for use with the teachings providedherein. Trademark 905 labels the source of the audio file, the title 910shows the user what music is contained in the audio file, selection 915is a function that takes the user to the next page in the graphicaldisplay, and credits 920 give proper attribution to producers,developers, and owners of the audio file.

The user will often have a set of multi-track digital audio files tolearn on the system. FIG. 10 illustrates a display for a graphical userinterface offering selections within a track list of a multi-track audiorecording set designed for use with the system for learning and mixingmusic, according to some embodiments. Track List Display 1000 shows theselections in a multi-track audio recording set or track list 1050. Theuser chooses an audio file and proceeds to the menu bar 1005 to select afunction from home page 1010, volume/fader page 1015, musictranscription and tablature 1020, looping page 1025, and help page 1030.

The user can access an information page regarding the piece of musicselected from the track list, such as the band, song title, album,tempo, and tuning. FIG. 11 illustrates a display for a graphical userinterface offering information on the track list selection in the systemfor learning and mixing music, according to some embodiments.Information Display 1100 provides the user with the selectionbibliographic information 1105 containing information on the band, songtitle, and album, and selection technical information 1110 providesinformation on the tempo and tuning for the selection. Play function1150 allows the user to begin learning the selection.

The volume/fader page 1015 has several functions that enables a user toeffectively learn and mix music. FIGS. 12-15 show variousfunctionalities of the volume/fader page 1015. FIG. 12 illustrates adisplay for a graphical user interface offering state selections fortrack faders, volume fader control, and transport selections, in whichall tracks are selected on a multi-track audio recording designed foruse with the system for learning and mixing music, according to someembodiments.

Volume/fader display 1200 provides the functionality of track on/off(i.e. mute) control 1205 for each track file. The functionality of thevolume/fader controls 1210 is provided by the volume indicators 1215 toindicate sound pressure level and the faders 1220,1225,1230,1235 toadjust volume in the manner of a potentiometer or digitometer, forexample. Transport section 1250 provides a time bar 1255 to indicate aposition in the piece of music and can also contain markings, such ascolors, to indicate the intro, pre-chorus, verse, solo, bridge, outro,chorus, and middle section, for example.

The transport section 1250 also provides several state selectionfunctions: a rewind 1260, pause 1265, fast forward 1270, stop 1275 asnormal transport control state settings; loop 1280 to allow a user torepeat a desired section of the piece of music; and slow 1285 to allowthe user to slow the song by a predetermined amount and serve the useras a function commonly known as a speed control, tempo adjustment, ortempo control. In some embodiments, the slow 1285 function can be adefault setting (e.g. to slow by some percentage, such as 50%, 75%, orthe like) and, in some embodiments, the user can define a desired speedsetting. As such, in some embodiments, the playback controls can be tothose found on about any tape deck or video cassette recorder, such as“previous”, “next”, “play”, “pause”, and “record”. And, in someembodiments the playback controls include a “looping” function to allowthe user to loop a particular section or measure over and over againuntil the user is satisfied in the learning experience.

In some embodiments, the transport section 1250 can be operable within asingle functioning section of a system, such as a single page of asoftware application. In some embodiments, the transport section 1250can be operable within a plurality of functioning sections of a system,such as in a plurality of pages of a software application, such that thetransporting can occur quickly at the choosing of the user withouthaving to move from one section of the system to another section of thesystem. A music label and timer 1290 is also provided as a referencedata point for the user.

In some embodiments, the mixing of audio can be handled on an individualsample per track basis. Each track can be represented individually,having to keep track of its own samples, duration, levels and peaks,gain, and time. Once each track can be initialized and loaded out of itsfile, and then handed off to a subsystem to decode the compression andmake samples. An example of a subsystem can include, for example Apple'sCOREAUDIO subsystem. After the samples are made available, the track canthen be assigned to a master track handler object referred to as themixer object, and saved recorded audio sessions can be loaded at thistime as well. The mixer object primes the subsystem and initializes theoutput. In these embodiments, the touching of the “play” button can beused to start the mixer in its function of combining the audio withinbuffers, where the mixer calls out to each track asking it for its nextframe of audio. The audio frames can be added to the playback buffer andenqueued, and all timing on the audio can then be synced to allow theaudio to be synced to a subnote level to reduce or eliminate creepwithin tracks.

In some embodiments, the audio recording can be handled by a mannersimilar to the individual audio playback, where extensions to record toa file and store its place within a song can be used. Once the userclicks a record function, a recorder object can be built, and the objectcan then start a file and initialize the recording. Once initialized,the recording class can store the current play time within the song tothe measure and begin. The user can tell the device to stop recording,and the object can then mark that time and store the duration of thedata stream in a settings file. The audio data is then flushed to thefile, a track handler object can then be created with its start time andduration set, and the mixer can be updated to allow future playback ofthe recorded audio along with the rest of the prerecorded audio.

FIG. 13 illustrates a display for a graphical user interface offeringstate selections for track faders, volume fader control, and transportselections, in which all tracks but the metronome track are selected ona multi-track audio recording designed for use with the system forlearning and mixing music, according to some embodiments. FIG. 14illustrates a display for a graphical user interface offering stateselections for track faders, volume fader control, and transportselections, in which only the isolated instrument audio track and theemulated instrument audio track are selected on a multi-track audiorecording designed for use with the system for learning and mixingmusic, according to some embodiments. FIG. 15 illustrates a display fora graphical user interface offering state selections for track faders,volume fader control, and transport selections, in which only theemulation audio track and the emulated instrument audio track areselected on a multi-track audio recording designed for use with thesystem for learning and mixing music, according to some embodiments.

A user can benefit by reading music while playing. FIG. 16 illustrates adisplay for a graphical user interface offering musical transcriptionand tablature audio data, in which only the current section of music isshown for a multi-track audio recording designed for use with the systemfor learning and mixing music, according to some embodiments. Musictranscription and tablature page 1600 provides the current transcriptionand tablature 1605 in a bright display and the upcoming transcriptionand tablature 1610 in a dim display, where the current represents themusic currently playing, and the upcoming represents the musicimmediately following the music currently playing. Each measure, forexample, can have a timecode and duration that represents where in thesong's timeline that measure is played. Using that information alongwith the song's current playback position, the tablature can besynchronized along with any playing audio. Each measure can be designedto display notes or chords and which strings or frets to use. In someembodiments, a user may desire a standard music scale for display. Themusical transcription and tablature can also be a dynamic, scrollingdisplay, in some embodiments.

A user can also benefit from a function that allows for an isolation ofa particular section of a piece of music, as well as a looping of thatsection to enable the user to focus and practice on that section of themusic. FIG. 17 illustrates a display for a graphical user interfaceoffering selection of a section of a piece of music by bar or set ofbars in a multi-track audio recording designed for use with the systemfor learning and mixing music, according to some embodiments. Loopingpage 1700 displays section 1705 of the digital audio file, and bar 1710or bars 1715, for example, can be isolated and selected for playback bythe user. The user can then use the methods taught herein to focus andlearn the particular section 1705, bar 1710, or bars 1715, for example.As such, in some embodiments, the playback can also be controlledthrough a “reel screen”, where every measure is segmented. In suchembodiments, any audio the user has recorded can also be displayed onthe screen in the measures in which it exists to allow the user toquickly find that audio and listen to their play of that session asgraphed over the original audio data. In some embodiments, a “tape” canbe graphically displayed to show the markings of the playback trackingbar.

The user may have questions, and as such, a help page is always useful.FIG. 18 illustrates a display for a graphical user interface offering ahelp page for the system for learning and mixing music, according tosome embodiments. Help page 1800 is a simple depiction of informationthat a user can obtain from the system.

FIG. 19 shows how a network may be used for the system for learning andmixing music, according to some embodiments. FIG. 19 shows severalcomputer systems coupled together through a network 1905, such as theinternet, along with a cellular network and related cellular devices.The term “internet” as used herein refers to a network of networks whichuses certain protocols, such as the TCP/IP protocol, and possibly otherprotocols such as the hypertext transfer protocol (HTTP) for hypertextmarkup language (HTML) documents that make up the world wide web (web).The physical connections of the internet and the protocols andcommunication procedures of the internet are well known to those ofskill in the art.

Access to the internet 1905 is typically provided by internet serviceproviders (ISP), such as the ISPs 1910 and 1915. Users on clientsystems, such as client computer systems 1930, 1950, and 1960 obtainaccess to the internet through the internet service providers, such asISPs 1910 and 1915. Access to the internet allows users of the clientcomputer systems to exchange information, receive and send e-mails, andview documents, such as documents which have been prepared in the HTMLformat. These documents are often provided by web servers, such as webserver 1920 which is considered to be “on” the internet. Often these webservers are provided by the ISPs, such as ISP 1910, although a computersystem can be set up and connected to the internet without that systemalso being an ISP.

The web server 1920 is typically at least one computer system whichoperates as a server computer system and is configured to operate withthe protocols of the world wide web and is coupled to the internet.Optionally, the web server 1920 can be part of an ISP which providesaccess to the internet for client systems. The web server 1920 is showncoupled to the server computer system 1925 which itself is coupled toweb content 1995, which can be considered a form of a media database.While two computer systems 1920 and 1925 are shown in FIG. 19, the webserver system 1920 and the server computer system 1925 can be onecomputer system having different software components providing the webserver functionality and the server functionality provided by the servercomputer system 1925 which will be described further below.

Cellular network interface 1943 provides an interface between a cellularnetwork and corresponding cellular devices 1944, 1946 and 1948 on oneside, and network 1905 on the other side. Thus cellular devices 1944,1946 and 1948, which may be personal devices including cellulartelephones, two-way pagers, personal digital assistants or other similardevices, may connect with network 1905 and exchange information such asemail, content, or HTTP-formatted data, for example. Cellular networkinterface 1943 is coupled to computer 1940, which communicates withnetwork 1905 through modem interface 1945. Computer 1940 may be apersonal computer, server computer or the like, and serves as a gateway.Thus, computer 1940 may be similar to client computers 1950 and 1960 orto gateway computer 1975, for example. Software or content may then beuploaded or downloaded through the connection provided by interface1943, computer 1940 and modem 1945.

Client computer systems 1930, 1950, and 1960 can each, with theappropriate web browsing software, view HTML pages provided by the webserver 1920. The ISP 1910 provides internet connectivity to the clientcomputer system 1930 through the modem interface 1935 which can beconsidered part of the client computer system 1930. The client computersystem can be a personal computer system, a network computer, a web TVsystem, or other such computer system.

Similarly, the ISP 1915 provides internet connectivity for clientsystems 1950 and 1960, although as shown in FIG. 19, the connections arenot the same as for more directly connected computer systems. Clientcomputer systems 1950 and 1960 are part of a LAN coupled through agateway computer 1975. While FIG. 19 shows the interfaces 1935 and 1945as generically as a “modem,” each of these interfaces can be an analogmodem, isdn modem, cable modem, satellite transmission interface (e.g.“direct PC”), or other interfaces for coupling a computer system toother computer systems.

Client computer systems 1950 and 1960 are coupled to a LAN 1970 throughnetwork interfaces 1955 and 1965, which can be ethernet network or othernetwork interfaces. The LAN 1970 is also coupled to a gateway computersystem 1975 which can provide firewall and other internet relatedservices for the local area network. This gateway computer system 1975is coupled to the ISP 1915 to provide internet connectivity to theclient computer systems 1950 and 1960. The gateway computer system 1975can be a conventional server computer system. Also, the web serversystem 1920 can be a conventional server computer system.

Alternatively, a server computer system 1980 can be directly coupled tothe LAN 1970 through a network interface 1985 to provide files 1990 andother services to the clients 1950, 1960, without the need to connect tothe internet through the gateway system 1975.

Through the use of such a network, for example, the system can alsoprovide an element of social networking, whereby users can contact otherusers having similar subject-profiles. In some embodiments, the systemcan include a messaging module operable to deliver notifications viaemail, SMS, and other mediums. In some embodiments, the system isaccessible through a portable, single unit device and, in someembodiments, the input device, the graphical user interface, or both, isprovided through a portable, single unit device. In some embodiments,the portable, single unit device is a hand-held device.

Regardless of the information presented, the system exemplifies thebroader concept of a system for learning and mixing music. The systemcan provide a powerful and unique learning experience and, in someembodiments, it can process multimedia in the form of text, images,video, and sound.

In some embodiments, the user can customize the system, such as choosinginterfaces, colors, language, music preferences and categories, etc. Theuser is allowed to enter preferences into the system in order tocustomize visual displays that present the information the user in apersonalized way. In some embodiments, the system includes a multimediainteraction of one or more of text and video; sound and diagrams,pictures, or images; sound; and video.

In some embodiments, the system and it's database can include any of avariety of system libraries that contain organized sets of any of avariety of information of value to users. Moreover, can information canbe obtained from external data sources, whereby plug-ins and APIs can bedesigned to allow integration with third party systems and exchange datawith external data sources. The external data sources can be used toprovide information on demand, to update existing information stored inthe system libraries, or both.

Some portions of the detailed description are presented in terms ofoperations of the system. The operations are those requiring physicalmanipulations of physical quantities resulting in a useful product beingproduced. In other words, a transformation is occurring, in someembodiments. In some embodiments, the transformation can be particularto the use of a particular machine or apparatus designed for thattransformation. Usually, though not necessarily, these quantities takethe form of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like. All of these and similar terms are to beassociated with the appropriate physical quantities and are merelyconvenient labels applied to these quantities. Unless specificallystated otherwise, discussions utilizing terms such as “processing” or“computing” or “calculating” or “determining” or “displaying” or thelike, refer to the action and processes of a computer system, or similarelectronic computing device, that manipulates and transforms datarepresented as physical (electronic) quantities within the computersystem's registers and memories into other data similarly represented asphysical quantities within the computer system memories or registers orother such information storage, transmission or display devices.

Moreover, the teachings relate to a system for performing the operationsherein. This system may be specially constructed as an apparatusdesigned solely for the required purposes, or it may comprise a generalpurpose computer selectively activated or reconfigured by a computerprogram stored in the computer. Such a computer program may be stored ina computer readable storage medium, such as, but is not limited to, anytype of disk including floppy disks, optical disks, CD-ROMs, andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any typeof media suitable for storing electronic instructions, and each coupledto a computer system bus.

It should be also appreciated that, in some embodiments, the methods anddisplays presented herein are not necessarily inherently related to anyparticular computer or other apparatus. Various general purpose systemsmay be used with programs in accordance with the teachings herein, or itmay prove convenient to construct more specialized apparatus to performthe methods of some embodiments. The required structure for a variety ofthese systems will be apparent to one of skill given the teachingsherein. In addition, the techniques are not described with reference toany particular programming language, and various embodiments may thus beimplemented using a variety of programming languages. Accordingly, theterms and examples provided above are illustrative only and not intendedto be limiting; and, the term “embodiment,” as used herein, means anembodiment that serves to illustrate by way of example and notlimitation. The following examples are illustrative of the uses of thepresent invention. It should be appreciated that the examples are forpurposes of illustration and are not to be construed as limiting to theinvention.

Example 1

In one aspect, a movie digital file is synchronized with and optionallyattached in the data structure with the audio tracks, such as forrecorded musical notation or any video that is played on a screen intime with the audio components. Another option is the inclusion of amusic XML file is possible such as for playing musical notation ifdesired or otherwise providing instructions for the playback of thework.

In a further embodiment, a graphical representation is stored in adevice that reproduces the musical notation of the part to be learned,and a visual display presents such musical notation in synchronism withthe audio. Preferably, the visual display provides a scrolling graphicalrepresentation, in which the musical notation is moved axially, orscrolled, along the display over time. A graphical marker in the displayarea indicates the present moment and the scrolling movement enables aplayer to visualize the imminent notes to be played. Preferably, thedisplay is configured to maximize the notation on the display of themusic currently played, making it visible from a distance. At the sametime, the amount of data visible at any one time is restricted to thedata that are strictly relevant with respect to the part of thecomposition played at that particular time and at times that shortlyprecede and follow the part being played.

In another embodiment, a display shows a visual representation of themusical notes to be played, which may be synchronized with the audioplayback so that the recent, present and imminent musical notes can beseen.

In one embodiment, a display provides an animated graphicalrepresentation of the musical notation relative to the part to belearned, whereby such graphical representation scrolls along the displayarea in synchronism with the audio music signal. In particular, a regionof the display area is marked to identify the present moment, with thegraphical representation of the music moving past the marks as the musicpart is being played. By this arrangement, the display area enables aplayer to see musical events in advance that are immediately followingthe present moment. In a variant of the present embodiment, recentmusical notation elements may also be seen in retrospect.

The apparatus and methods of presenting musical notation areadvantageous because a student is provided with a temporal context forthe musical events of interest by presenting the notation in a linearform. Contrary to traditional sheet music notation, the scrolling of themusic can be continuous, removing the need for page turns.

In another embodiment, apparatus and methods are also provided forannotating musical events within the graphical score to ensure asynchronization of the visual display with the audio musical output.This synchronization may be generated manually or automatically andinvolves associating one or more points in the graphical representationwith the corresponding time at which they should be displayed. Thedisplay control system adjusts the presentation rate of the graphicaldata so that the visual representation is aligned with the audio signalat points where visual representation and audio signals have beenassociated previously. Alternatively, the rate of progress may beadjusted throughout the graphical representation to ensure thatsynchronization can be done in the production stage of the graphicalmaterial. In the latter case, any one of the known methods of graphicalinterpolation may be used to adjust the rate at which the graphicalvisualization proceeds, so that the processed version can be played backat a fixed rate but with the assurance of being aligned with the audio.

In order to ensure that displayed musical notation is synchronized withthe audio, presentation time stamps can be encoded with the notationdata, whether the notation is stored as a rendered video representationor as raw data requiring rendering to a visual form. The rate at whichthe visual notation is presented may be controlled by the device, so toensure synchronism with the audio. In particular, the rate at which thevisual notation is presented is adjusted on the basis of an estimatederror between it and the audio at the current rate of presentation ofeach. If it is estimated that a point in the notation associated with apresentation time stamp will be presented after the corresponding pointin the audio tracks, then the presentation rate for the notation isincreased. Conversely, if the estimated presentation time for a timestamped point in the visual notation is early relative to the audiotracks, then the presentation rate for the notation can be slowed. Theamount by which the presentation rate is adjusted can be derived fromthe degree of estimated presentation time error.

If synchronization of the graphical material is to be performed in aplayback mode, the playback device must be able to advance through thegraphical material at a variable rate. To this end, the frame rate for avideo presentation may be changed (if the display mechanism can tolerateinconsistent frame durations) by interpolating new frames from thestored frames during playback, or by altering the rate of rendering ofthe graphical elements in a graphical representation, which—unlikevideo—during audio playback is rendered from a stored formalizedrepresentation.

Adjustments to the presentation rate to achieve synchronism areimplemented by determining the timing error that will arise to the nextmarked graphical point at the current playback rate. Such determinationmay be performed through a dedicated component. If the next markedgraphical point occurs late, the presentation rate is increased, or ifit occurs early, the presentation rate is reduced. This error isassessed periodically and the presentation rate adjusted accordingly sothat the actual timing error is minimal and unnoticeable. When a markedpoint is passed, the process continues using the next marked point or,in the absence of another mark, by using the end of the graphicalpresentation and the end of the audio recording. This arrangement isparticularly advantageous because audio and visual synchronism areinsured despite variations in the musical tempo, which frequently occuras an artifact of human performance and also as an artistic devicewithin a composition.

A video file (for example, a MP4 movie file) is then attached andsynchronized with the above described audio file. This video file mayinclude, among other things, a video display of the musical notations orother visual instructions. A music XML file may also be attached andsynchronized with the above described audio files, for example, toprovide the musical notation.

The audio tracks may be encoded into a digital file format using acontainer method, in order to support several separate but synchronizedaudio channels. The RIFF file formats, such as wav and avi are examplesof this type of file. Additionally, since certain of these file formatsare configured to allow more than one data type to be stored together,video or notation data may also be included. A single data file may theninclude all of the information required to play the audio, present thevisual notation and to record user performances. Other relevant data mayalso be included, for example musical instrument digital interface(MIDI) data and lyrics.

We claim:
 1. A video game console system for mixing complex multimediadata using audio tempo mapping, the system comprising: a processor and amemory on a non-transitory computer readable medium, the memoryincluding a multi-track digital file of a preselected performance havinga plurality of audio tracks that include an isolated instrument audiotrack containing only audio from an isolated instrument havingvariations in musical tempo, and a digital image file that issynchronized with the variations in the musical tempo; and, a variabletiming reference track designed and provided for a user for thepreselected performance, wherein the preselected performance wasprerecorded, and the designing of the variable timing reference trackincludes creating a tempo map having variable tempos, rhythms, and animage display module on a non-transitory computer readable storagemedium and in operable communication with the processor for viewing thedigital images synchronized to the variations in the musical tempo usingthe variable timing reference track.
 2. The system of claim 1 furthercomprising: an integration engine on a non-transitory computer readablestorage medium and in operable communication with the processor forcombining an emulated performance with the isolated instrument audiotrack, the emulation audio track, the variable timing reference track,or a combination thereof, to transform the multi-track digital file intothe audio file that is synchronized with the digital images using thevariations in the musical tempo through the variable timing referencetrack.
 3. The system of claim 1, wherein the system includes notesconfigured as MIDI notes that were manually created using transients. 4.The system of claim 1, wherein the digital image includes musictranscription and tablature data, wherein the music transcription andtablature data is synchronized to the preselected performance throughthe variable timing reference track.
 5. The system of claim 1, whereinthe multi-track digital audio file of the preselected performanceincludes an original, multi-track recording or a re-creation of theoriginal, multi-track recording having variable tempos in theperformance.
 6. The system of claim 1, wherein the system furtherincludes a transformation module on a non-transitory computer readablestorage medium and in operable communication with the processor fortransforming the multi-track digital audio file to include a variablegain ratio of (i) the isolated instrument audio track to (ii) anemulation audio track, wherein the emulation audio track represents asubtraction of the isolated instrument audio track from the plurality ofaudio tracks, and the gain ratio is selected by the user; and, anemulation module on a non-transitory computer readable storage mediumand in operable communication with the processor for emulating theperformance under a guidance of the variable timing reference track; theplurality of audio tracks further include an emulation audio trackconsisting of the multi-track digital audio file having the digitalimage file removed; and, the digital images further comprise one or morebars of the isolated instrument audio track selected to enable the userto focus on emulating a section of the preselected performance.
 7. Thesystem of claim 1 including a transport state selection, wherein thedigital image includes a looping page showing a bar or bars of thepreselected performance for repeated playback, wherein the bar or barsare selected by the user for the repeated playback through the transportstate selection, the transport state selection configured to indicate aposition in the performance according to the variable timing referencetrack.
 8. The system of claim 7, wherein the transport state selectionis configured to include tempo speed control to speed up and slow downthe variable tempo performance as desired.
 9. The system of claim 1,further comprising a data exchange module on a non-transitory computerreadable storage medium and in operable communication with the processorfor exchanging data with an external non-transitory computer readablestorage media.
 10. The system of claim 1, wherein the system is on ahandheld device.
 11. The system of claim 10, wherein the device is acellular device.
 12. The system of 1, wherein the digital image file isconfigured for synchronizing with the variations in the musical tempo,the digital image file configured with a frame rate for a videopresentation that synchronizes with the audio during playback byinterpolating new frames from stored frames; or, altering the rate ofrendering of the graphical elements in a graphical representation.
 13. Avideo game console system for providing users with an opportunity forgaming that includes variable tempo audio through viewing imagessynchronized to the variable tempo audio, comprising: a networkinterface; and, a server computer system having a processor in operableconnection with a web server on a non-transitory computer readablestorage medium, the web server having custom audio files accessible to auser, the audio files including a multi-track digital file of apreselected performance having a plurality of audio tracks that includean isolated instrument audio track containing only audio from anisolated instrument having variations in musical tempo, and a digitalimage file that is synchronized with the variations in the musicaltempo; and, a variable timing reference track designed and provided fora user for the preselected performance, wherein the preselectedperformance was prerecorded, and the variable timing reference trackincludes a tempo map having variable tempos, rhythms, and beats usingnotes from the preselected performance; wherein, the system provides theuser with the opportunity for gaming in synchronization with thevariable tempos, rhythms, and beats of the isolated instrument audiotrack.
 14. The system of claim 13 further comprising the system of claim2 and further configured with an emulation recording module on anon-transitory computer readable storage medium for the user to recordan emulated performance on a non-transitory computer readable storagemedium; combine the emulated performance with the isolated instrumentaudio track, the emulation audio track, the variable timing referencetrack, or a combination thereof, to transform the multi-track digitalaudio file into an educational performance file; view the educationalperformance file; and, repeat the emulating, recording, combining, andviewing until the user has learned the performance to the user'ssatisfaction; wherein, the multi-track digital audio file includes anemulation audio track consisting of the multi-track digital audio filehaving the digital image file removed.
 15. The system of claim 13,wherein the web server has a multi-track digital audio file of thepreselected performance which is an original, multi-track recording or are-creation of the original, multi-track recording having variabletempos in the performance.
 16. The system of claim 13, wherein theemulating includes viewing the web server includes custom audio filesconfigured to include a looping page showing a bar or bars of thepreselected performance for repeated playback, wherein the bar or barsare selected by the user for the repeated playback through a transportstate selection, the transport state selection configured to indicate aposition in the performance according to the variable timing referencetrack.